{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fa3b0e3c",
   "metadata": {},
   "source": [
    "# Build a Milvus Powered Text-Image Search Engine in Minutes\n",
    "\n",
    "This notebook illustrates how to build an text-image search engine from scratch using [Milvus](https://milvus.io/). Milvus is the most advanced open-source vector database built for AI applications and supports nearest neighbor embedding search across tens of millions of entries. We'll go through text-image search procedures and evaluate the performance. Moreover, we managed to make the core functionality as simple as a dozen lines of code, with which you can start hacking your own image search engine."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97a009a7",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "### Install Dependencies\n",
    "First we need to install dependencies such as towhee, gradio and opencv-python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3986e8d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m pip install -q towhee gradio opencv-python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11ef6b1a",
   "metadata": {},
   "source": [
    "### Prepare the data\n",
    "\n",
    "The dataset used in this demo is a subset of the ImageNet dataset (100 classes, 10 images for each class), and the dataset is available via [Github](https://github.com/towhee-io/examples/releases/download/data/reverse_image_search.zip). \n",
    "\n",
    "The dataset is organized as follows:\n",
    "- **train**: directory of candidate images;\n",
    "- **test**: directory of test images;\n",
    "- **reverse_image_search.csv**: a csv file containing an ***id***, ***path***, and ***label*** for each image;\n",
    "\n",
    "Let's take a quick look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "daf4abc2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0\n",
      "100  119M  100  119M    0     0  8071k      0  0:00:15  0:00:15 --:--:-- 10.6M\n"
     ]
    }
   ],
   "source": [
    "! curl -L https://github.com/towhee-io/examples/releases/download/data/reverse_image_search.zip -O\n",
    "! unzip -q -o reverse_image_search.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "490d1379",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>path</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>./train/brain_coral/n01917289_1783.JPEG</td>\n",
       "      <td>brain_coral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>./train/brain_coral/n01917289_4317.JPEG</td>\n",
       "      <td>brain_coral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>./train/brain_coral/n01917289_765.JPEG</td>\n",
       "      <td>brain_coral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>./train/brain_coral/n01917289_1079.JPEG</td>\n",
       "      <td>brain_coral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>./train/brain_coral/n01917289_2484.JPEG</td>\n",
       "      <td>brain_coral</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                     path        label\n",
       "0   0  ./train/brain_coral/n01917289_1783.JPEG  brain_coral\n",
       "1   1  ./train/brain_coral/n01917289_4317.JPEG  brain_coral\n",
       "2   2   ./train/brain_coral/n01917289_765.JPEG  brain_coral\n",
       "3   3  ./train/brain_coral/n01917289_1079.JPEG  brain_coral\n",
       "4   4  ./train/brain_coral/n01917289_2484.JPEG  brain_coral"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('reverse_image_search.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2171f2e7",
   "metadata": {},
   "source": [
    "To use the dataset for text-image search, let's first define some helper function:\n",
    "\n",
    "- **read_images(results)**: read images by image IDs;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dd1b0ef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "from towhee.types.image import Image\n",
    "\n",
    "id_img = df.set_index('id')['path'].to_dict()\n",
    "def read_images(results):\n",
    "    imgs = []\n",
    "    for re in results:\n",
    "        path = id_img[re.id]\n",
    "        imgs.append(Image(cv2.imread(path), 'BGR'))\n",
    "    return imgs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3e98f62",
   "metadata": {},
   "source": [
    "### Create a Milvus Collection\n",
    "\n",
    "Before getting started, please make sure that you have started a [Milvus service](https://milvus.io/docs/install_standalone-docker.md). This notebook uses [milvus 2.2.10](https://milvus.io/docs/v2.2.x/install_standalone-docker.md) and [pymilvus 2.2.11](https://milvus.io/docs/release_notes.md#2210)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f36a6f77",
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m pip install -q pymilvus==2.2.11"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc6de57b",
   "metadata": {},
   "source": [
    "Let's first create a `text_image_search` collection that uses the [L2 distance metric](https://milvus.io/docs/metric.md#Euclidean-distance-L2) and an [IVF_FLAT index](https://milvus.io/docs/index.md#IVF_FLAT)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "09b28770",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection, utility\n",
    "\n",
    "def create_milvus_collection(collection_name, dim):\n",
    "    connections.connect(host='127.0.0.1', port='19530')\n",
    "    \n",
    "    if utility.has_collection(collection_name):\n",
    "        utility.drop_collection(collection_name)\n",
    "    \n",
    "    fields = [\n",
    "    FieldSchema(name='id', dtype=DataType.INT64, descrition='ids', is_primary=True, auto_id=False),\n",
    "    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, descrition='embedding vectors', dim=dim)\n",
    "    ]\n",
    "    schema = CollectionSchema(fields=fields, description='text image search')\n",
    "    collection = Collection(name=collection_name, schema=schema)\n",
    "\n",
    "    # create IVF_FLAT index for collection.\n",
    "    index_params = {\n",
    "        'metric_type':'L2',\n",
    "        'index_type':\"IVF_FLAT\",\n",
    "        'params':{\"nlist\":512}\n",
    "    }\n",
    "    collection.create_index(field_name=\"embedding\", index_params=index_params)\n",
    "    return collection\n",
    "\n",
    "collection = create_milvus_collection('text_image_search', 512)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a30aa33c",
   "metadata": {},
   "source": [
    "## Text Image Search\n",
    "\n",
    "In this section, we'll show how to build our text-image search engine using Milvus. The basic idea behind our text-image search is the extract embeddings from images and texts using a deep neural network and compare the embeddings with those stored in Milvus.\n",
    "\n",
    "We use [Towhee](https://towhee.io/), a machine learning framework that allows for creating data processing pipelines, and it also provides predefined operators which implement insert and query operation in Milvus.\n",
    "\n",
    "<img src=\"./workflow.png\" width = \"60%\" height = \"60%\" align=center />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ee02951",
   "metadata": {},
   "source": [
    "### Generate image and text embeddings with CLIP\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eefdd5e5",
   "metadata": {},
   "source": [
    "This operator extracts features for image or text with [CLIP](https://openai.com/blog/clip/) which can generate embeddings for text and image by jointly training an image encoder and text encoder to maximize the cosine similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1410b36c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from towhee import ops, pipe, DataCollection\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "54927fb3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-12 14:54:53,081 - 140406106779840 - connectionpool.py-connectionpool:1003 - DEBUG: Starting new HTTPS connection (1): huggingface.co:443\n",
      "2023-06-12 14:54:54,529 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:54:56,725 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/vocab.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:54:57,253 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/preprocessor_config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:54:57,544 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/tokenizer_config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:54:57,626 - 140397301311232 - node.py-node:167 - INFO: Begin to run Node-_input\n",
      "2023-06-12 14:54:57,629 - 140397301311232 - node.py-node:167 - INFO: Begin to run Node-image-decode/cv2-0\n",
      "2023-06-12 14:54:57,630 - 140397292918528 - node.py-node:167 - INFO: Begin to run Node-image-text-embedding/clip-1\n",
      "2023-06-12 14:54:57,631 - 140397284525824 - node.py-node:167 - INFO: Begin to run Node-lambda-2\n",
      "2023-06-12 14:54:57,633 - 140397276133120 - node.py-node:167 - INFO: Begin to run Node-_output\n",
      "2023-06-12 14:54:57,814 - 140406106779840 - __init__.py-__init__:307 - DEBUG: matplotlib data path: /home/junjie.jiangjjj/anaconda3/envs/py39/lib/python3.9/site-packages/matplotlib/mpl-data\n",
      "2023-06-12 14:54:57,821 - 140406106779840 - __init__.py-__init__:307 - DEBUG: CONFIGDIR=/home/junjie.jiangjjj/.config/matplotlib\n",
      "2023-06-12 14:54:57,823 - 140406106779840 - __init__.py-__init__:1475 - DEBUG: interactive is False\n",
      "2023-06-12 14:54:57,824 - 140406106779840 - __init__.py-__init__:1476 - DEBUG: platform is linux\n",
      "2023-06-12 14:54:57,877 - 140406106779840 - __init__.py-__init__:307 - DEBUG: CACHEDIR=/home/junjie.jiangjjj/.cache/matplotlib\n",
      "2023-06-12 14:54:57,880 - 140406106779840 - font_manager.py-font_manager:1540 - DEBUG: Using fontManager instance from /home/junjie.jiangjjj/.cache/matplotlib/fontlist-v330.json\n",
      "2023-06-12 14:54:58,069 - 140406106779840 - pyplot.py-pyplot:339 - DEBUG: Loaded backend module://matplotlib_inline.backend_inline version unknown.\n",
      "2023-06-12 14:54:58,070 - 140406106779840 - pyplot.py-pyplot:339 - DEBUG: Loaded backend module://matplotlib_inline.backend_inline version unknown.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">img</th> <th style=\"text-align: center; font-size: 130%; border: none;\">vec</th></tr>\n",
       "<tr><td style=\"text-align: center; vertical-align: top; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \"><img src=\"\" width = \"128px\" height = \"128px\" style = \"float:left; padding:2px\"></td><td style=\"text-align: left; vertical-align: top; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">[0.037240546, -0.065988705, -0.010860455, ...] shape=(512,)</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 128x128 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "p = (\n",
    "    pipe.input('path')\n",
    "    .map('path', 'img', ops.image_decode.cv2('rgb'))\n",
    "    .map('img', 'vec', ops.image_text_embedding.clip(model_name='clip_vit_base_patch16', modality='image'))\n",
    "    .map('vec', 'vec', lambda x: x / np.linalg.norm(x))\n",
    "    .output('img', 'vec')\n",
    ")\n",
    "\n",
    "DataCollection(p('./teddy.png')).show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bdbea7e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-12 14:55:03,300 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:55:05,445 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/vocab.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:55:05,830 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/preprocessor_config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:55:06,123 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/tokenizer_config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:55:06,205 - 140396121286400 - node.py-node:167 - INFO: Begin to run Node-_input\n",
      "2023-06-12 14:55:06,207 - 140396121286400 - node.py-node:167 - INFO: Begin to run Node-image-text-embedding/clip-0\n",
      "2023-06-12 14:55:06,211 - 140394427840256 - node.py-node:167 - INFO: Begin to run Node-lambda-1\n",
      "2023-06-12 14:55:06,211 - 140394419447552 - node.py-node:167 - INFO: Begin to run Node-_output\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">text</th> <th style=\"text-align: center; font-size: 130%; border: none;\">vec</th></tr>\n",
       "<tr><td style=\"text-align: center; vertical-align: top; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">A teddybear on a skateboard in Times Square.</td><td style=\"text-align: left; vertical-align: top; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">[-0.0086854, 0.02717687, -0.0007425508, ...] shape=(512,)</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "p2 = (\n",
    "    pipe.input('text')\n",
    "    .map('text', 'vec', ops.image_text_embedding.clip(model_name='clip_vit_base_patch16', modality='text'))\n",
    "    .map('vec', 'vec', lambda x: x / np.linalg.norm(x))\n",
    "    .output('text', 'vec')\n",
    ")\n",
    "\n",
    "DataCollection(p2(\"A teddybear on a skateboard in Times Square.\")).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd931463",
   "metadata": {},
   "source": [
    "Here is detailed explanation of the code:\n",
    "\n",
    "- `map('path', 'img', ops.image_decode.cv2_rgb('rgb'))`: for each row from the data, read and decode the image at `path` and put the pixel data into column `img`;\n",
    "\n",
    "- `map('img', 'vec', ops.image_text_embedding.clip(model_name='clip_vit_base_patch16',modality='image'/'text')`: extract image or text embedding feature with `ops.image_text_embedding.clip`, an operator from the [Towhee hub](https://towhee.io/image-text-embedding/clip) . This operator supports seveal models including `clip_vit_base_patch16`,`clip_vit_base_patch32`,`clip_vit_large_patch14`,`clip_vit_large_patch14_336`,etc."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61433908",
   "metadata": {},
   "source": [
    "### Load Image Embeddings into Milvus\n",
    "\n",
    "We first extract embeddings from images with `clip_vit_base_patch16` model and insert the embeddings into Milvus for indexing. Towhee provides a [method-chaining style API](https://towhee.readthedocs.io/en/main/index.html) so that users can assemble a data processing pipeline with operators. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0c70df43",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-12 16:16:47,531 - 140406106779840 - connectionpool.py-connectionpool:273 - DEBUG: Resetting dropped connection: huggingface.co\n",
      "2023-06-12 16:16:48,943 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 16:16:51,909 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/vocab.json HTTP/1.1\" 200 0\n",
      "2023-06-12 16:16:52,316 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/preprocessor_config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 16:16:52,605 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/tokenizer_config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 16:16:52,733 - 140381920413440 - node.py-node:167 - INFO: Begin to run Node-_input\n",
      "2023-06-12 16:16:52,748 - 140381912020736 - node.py-node:167 - INFO: Begin to run Node-read_csv-0\n",
      "2023-06-12 16:16:52,750 - 140402136966912 - node.py-node:167 - INFO: Begin to run Node-image-decode/cv2-1\n",
      "2023-06-12 16:16:52,750 - 140381920413440 - node.py-node:167 - INFO: Begin to run Node-image-text-embedding/clip-2\n",
      "2023-06-12 16:16:52,752 - 140386089555712 - node.py-node:167 - INFO: Begin to run Node-lambda-3\n",
      "2023-06-12 16:16:52,753 - 140393303758592 - node.py-node:167 - INFO: Begin to run Node-ann-insert/milvus-client-4\n",
      "2023-06-12 16:16:52,755 - 140393043715840 - node.py-node:167 - INFO: Begin to run Node-_output\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 5s, sys: 46.7 s, total: 1min 52s\n",
      "Wall time: 28.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "collection = create_milvus_collection('text_image_search', 512)\n",
    "\n",
    "def read_csv(csv_path, encoding='utf-8-sig'):\n",
    "    import csv\n",
    "    with open(csv_path, 'r', encoding=encoding) as f:\n",
    "        data = csv.DictReader(f)\n",
    "        for line in data:\n",
    "            yield int(line['id']), line['path']\n",
    "\n",
    "p3 = (\n",
    "    pipe.input('csv_file')\n",
    "    .flat_map('csv_file', ('id', 'path'), read_csv)\n",
    "    .map('path', 'img', ops.image_decode.cv2('rgb'))\n",
    "    .map('img', 'vec', ops.image_text_embedding.clip(model_name='clip_vit_base_patch16', modality='image', device=0))\n",
    "    .map('vec', 'vec', lambda x: x / np.linalg.norm(x))\n",
    "    .map(('id', 'vec'), (), ops.ann_insert.milvus_client(host='127.0.0.1', port='19530', collection_name='text_image_search'))\n",
    "    .output()\n",
    ")\n",
    "\n",
    "ret = p3('reverse_image_search.csv')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "471627b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0c27fb6c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of inserted data is 0.\n"
     ]
    }
   ],
   "source": [
    "print('Total number of inserted data is {}.'.format(collection.num_entities))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1b1538a",
   "metadata": {},
   "source": [
    "### Query Matched Images from Milvus"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e04eff7",
   "metadata": {},
   "source": [
    "Now that embeddings for candidate images have been inserted into Milvus, we can query across it for nearest neighbors. Again, we use Towhee to load the input Text, compute an embedding vector, and use the vector as a query for Milvus. Because Milvus only outputs image IDs and distance values, we provide a `read_images` function to get the original image based on IDs and display."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "02483f5a",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-12 14:56:15,172 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:56:17,239 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/vocab.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:56:17,616 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/preprocessor_config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:56:17,946 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/tokenizer_config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 14:56:18,058 - 140385418467072 - node.py-node:167 - INFO: Begin to run Node-_input\n",
      "2023-06-12 14:56:18,059 - 140385391195904 - node.py-node:167 - INFO: Begin to run Node-image-text-embedding/clip-0\n",
      "2023-06-12 14:56:18,061 - 140385382803200 - node.py-node:167 - INFO: Begin to run Node-lambda-1\n",
      "2023-06-12 14:56:18,061 - 140384248522496 - node.py-node:167 - INFO: Begin to run Node-ann-search/milvus-client-2\n",
      "2023-06-12 14:56:18,062 - 140385418467072 - node.py-node:167 - INFO: Begin to run Node-lambda-3\n",
      "2023-06-12 14:56:18,063 - 140384240129792 - node.py-node:167 - INFO: Begin to run Node-read_image-4\n",
      "2023-06-12 14:56:18,065 - 140383858194176 - node.py-node:167 - INFO: Begin to run Node-_output\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">text</th> <th style=\"text-align: center; font-size: 130%; border: none;\">images</th></tr>\n",
       "<tr><td style=\"text-align: center; vertical-align: top; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">A white dog</td><td style=\"text-align: center; vertical-align: top; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \"><img src=\"\" width = \"128px\" height = \"128px\" style = \"float:left; padding:2px\"> <img src=\"\" width = \"128px\" height = \"128px\" style = \"float:left; padding:2px\"> <img src=\"\" width = \"128px\" height = \"128px\" style = \"float:left; padding:2px\"> <img src=\"\" width = \"128px\" height = \"128px\" style = \"float:left; padding:2px\"> <img src=\"\" width = \"128px\" height = \"128px\" style = \"float:left; padding:2px\"></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-12 14:56:18,246 - 140393312151296 - node.py-node:167 - INFO: Begin to run Node-_input\n",
      "2023-06-12 14:56:18,247 - 140385391195904 - node.py-node:167 - INFO: Begin to run Node-image-text-embedding/clip-0\n",
      "2023-06-12 14:56:18,247 - 140385382803200 - node.py-node:167 - INFO: Begin to run Node-lambda-1\n",
      "2023-06-12 14:56:18,247 - 140384248522496 - node.py-node:167 - INFO: Begin to run Node-ann-search/milvus-client-2\n",
      "2023-06-12 14:56:18,248 - 140385418467072 - node.py-node:167 - INFO: Begin to run Node-lambda-3\n",
      "2023-06-12 14:56:18,248 - 140384240129792 - node.py-node:167 - INFO: Begin to run Node-read_image-4\n",
      "2023-06-12 14:56:18,248 - 140383858194176 - node.py-node:167 - INFO: Begin to run Node-_output\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">text</th> <th style=\"text-align: center; font-size: 130%; border: none;\">images</th></tr>\n",
       "<tr><td style=\"text-align: center; vertical-align: top; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">A black dog</td><td style=\"text-align: center; vertical-align: top; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \"><img src=\"\" width = \"128px\" height = \"128px\" style = \"float:left; padding:2px\"> <img src=\"\" width = \"128px\" height = \"128px\" style = \"float:left; padding:2px\"> <img src=\"\" width = \"128px\" height = \"128px\" style = \"float:left; padding:2px\"> <img src=\"\" width = \"128px\" height = \"128px\" style = \"float:left; padding:2px\"> <img src=\"\" width = \"128px\" height = \"128px\" style = \"float:left; padding:2px\"></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import cv2\n",
    "\n",
    "def read_image(image_ids):\n",
    "    df = pd.read_csv('reverse_image_search.csv')\n",
    "    id_img = df.set_index('id')['path'].to_dict()\n",
    "    imgs = []\n",
    "    decode = ops.image_decode.cv2('rgb')\n",
    "    for image_id in image_ids:\n",
    "        path = id_img[image_id]\n",
    "        imgs.append(decode(path))\n",
    "    return imgs\n",
    "\n",
    "\n",
    "p4 = (\n",
    "    pipe.input('text')\n",
    "    .map('text', 'vec', ops.image_text_embedding.clip(model_name='clip_vit_base_patch16', modality='text'))\n",
    "    .map('vec', 'vec', lambda x: x / np.linalg.norm(x))\n",
    "    .map('vec', 'result', ops.ann_search.milvus_client(host='127.0.0.1', port='19530', collection_name='text_image_search', limit=5))\n",
    "    .map('result', 'image_ids', lambda x: [item[0] for item in x])\n",
    "    .map('image_ids', 'images', read_image)\n",
    "    .output('text', 'images')\n",
    ")\n",
    "\n",
    "DataCollection(p4(\"A white dog\")).show()\n",
    "DataCollection(p4(\"A black dog\")).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df6f54b8",
   "metadata": {},
   "source": [
    "## Release a Showcase"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea9ec2e3",
   "metadata": {},
   "source": [
    "We've done an excellent job on the core functionality of our text-image search engine. Now it's time to build a showcase with interface. [Gradio](https://gradio.app/) is a great tool for building demos. With Gradio, we simply need to wrap the data processing pipeline via a `search_in_milvus` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c63fb53b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-12 16:22:30,654 - 140406106779840 - connectionpool.py-connectionpool:273 - DEBUG: Resetting dropped connection: huggingface.co\n",
      "2023-06-12 16:22:32,024 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 16:22:34,339 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/vocab.json HTTP/1.1\" 200 0\n",
      "2023-06-12 16:22:34,717 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/preprocessor_config.json HTTP/1.1\" 200 0\n",
      "2023-06-12 16:22:35,002 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://huggingface.co:443 \"HEAD /openai/clip-vit-base-patch16/resolve/main/tokenizer_config.json HTTP/1.1\" 200 0\n"
     ]
    }
   ],
   "source": [
    "search_pipeline = (\n",
    "    pipe.input('text')\n",
    "    .map('text', 'vec', ops.image_text_embedding.clip(model_name='clip_vit_base_patch16', modality='text'))\n",
    "    .map('vec', 'vec', lambda x: x / np.linalg.norm(x))\n",
    "    .map('vec', 'result', ops.ann_search.milvus_client(host='127.0.0.1', port='19530', collection_name='text_image_search', limit=5))\n",
    "    .map('result', 'image_ids', lambda x: [item[0] for item in x])\n",
    "    .output('image_ids')\n",
    ")\n",
    "\n",
    "def search(text):\n",
    "    df = pd.read_csv('reverse_image_search.csv')\n",
    "    id_img = df.set_index('id')['path'].to_dict()\n",
    "    imgs = []\n",
    "    image_ids = search_pipeline(text).to_list()[0][0]\n",
    "    return [id_img[image_id] for image_id in image_ids]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5204bce3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-12 16:22:37,627 - 140392909498112 - connectionpool.py-connectionpool:1003 - DEBUG: Starting new HTTPS connection (1): api.gradio.app:443\n",
      "2023-06-12 16:22:37,629 - 140406106779840 - connectionpool.py-connectionpool:1003 - DEBUG: Starting new HTTPS connection (1): api.gradio.app:443\n",
      "2023-06-12 16:22:39,054 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://api.gradio.app:443 \"GET /pkg-version HTTP/1.1\" 200 21\n",
      "2023-06-12 16:22:39,128 - 140402095003392 - selector_events.py-selector_events:54 - DEBUG: Using selector: EpollSelector\n",
      "2023-06-12 16:22:39,134 - 140406106779840 - connectionpool.py-connectionpool:228 - DEBUG: Starting new HTTP connection (1): 127.0.0.1:7863\n",
      "2023-06-12 16:22:39,138 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: http://127.0.0.1:7863 \"GET /startup-events HTTP/1.1\" 200 5\n",
      "2023-06-12 16:22:39,143 - 140406106779840 - connectionpool.py-connectionpool:228 - DEBUG: Starting new HTTP connection (1): 127.0.0.1:7863\n",
      "2023-06-12 16:22:39,153 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: http://127.0.0.1:7863 \"HEAD / HTTP/1.1\" 200 0\n",
      "2023-06-12 16:22:39,157 - 140406106779840 - connectionpool.py-connectionpool:1003 - DEBUG: Starting new HTTPS connection (1): api.gradio.app:443\n",
      "2023-06-12 16:22:39,174 - 140392909498112 - connectionpool.py-connectionpool:456 - DEBUG: https://api.gradio.app:443 \"POST /gradio-initiated-analytics/ HTTP/1.1\" 200 None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on local URL:  http://127.0.0.1:7863\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-12 16:22:42,604 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://api.gradio.app:443 \"GET /v2/tunnel-request HTTP/1.1\" 200 None\n",
      "2023-06-12 16:22:44,331 - 140406106779840 - connectionpool.py-connectionpool:1003 - DEBUG: Starting new HTTPS connection (1): 0353404c1d46f8b38d.gradio.live:443\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on public URL: https://0353404c1d46f8b38d.gradio.live\n",
      "\n",
      "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-12 16:22:46,117 - 140406106779840 - connectionpool.py-connectionpool:456 - DEBUG: https://0353404c1d46f8b38d.gradio.live:443 \"HEAD / HTTP/1.1\" 200 0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><iframe src=\"https://0353404c1d46f8b38d.gradio.live\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-12 16:22:46,133 - 140403532490496 - connectionpool.py-connectionpool:1003 - DEBUG: Starting new HTTPS connection (1): api.gradio.app:443\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-12 16:22:47,753 - 140403532490496 - connectionpool.py-connectionpool:456 - DEBUG: https://api.gradio.app:443 \"POST /gradio-launched-telemetry/ HTTP/1.1\" 200 None\n"
     ]
    }
   ],
   "source": [
    "import gradio\n",
    "\n",
    "interface = gradio.Interface(search, \n",
    "                             gradio.inputs.Textbox(lines=1),\n",
    "                             [gradio.outputs.Image(type=\"filepath\", label=None) for _ in range(5)]\n",
    "                            )\n",
    "\n",
    "interface.launch(inline=True, share=True)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "452ee1ccfaba3d4bb36278ef299934b47178eea3ed5a06471ad5391f6c0e0271"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
